Estimating Reliability for Each Study

The current paper describes and illustrates three things: (a) sampling variance and confidence intervals for Cronbach’s Alpha, (b) the relative precision of reliability estimates from local studies and meta-analyses, and (c) how to blend the local and metaanalytic information to create an optimal local reliability estimate according to Bayesian principles. The paper is not about artifact corrections used to compute a meta-analysis. Rather it is about using information contained in a meta-analysis to improve local estimates of reliability. The improved estimates can result in better estimates and corrections for artifacts at the local level. Estimating Reliability 3 Estimating Reliability in Primary Research Measurement experts routinely call for the estimation of the reliability of all measures (scores) used in a study based on that study’s data (e.g., Thompson, 2003; Whittington, 1998). That is, primary researchers are asked to report estimates of the reliability of their measures based on their data. Despite such calls for reporting local estimates, many researchers fail to report any reliability estimates at all or else simply report estimates taken from test manuals or other literature reporting the development of the measure (e.g., Vacha-Haase, Ness, Nilsson, & Reetz, 1999, Yin & Fan, 2000). Measurement experts note that reliability estimates reported in test manuals or in articles reporting the development of a measure may not adequately represent the reliability of the data in any particular study because of the influence of the research context, including the variability of the trait in the population of interest and the context of measurement, including such factors as the purpose of measurement (e.g., selection vs. developmental feedback) and the testing conditions (e.g., noise, light, time of day, etc.). On the other hand, such calls for local reliability estimates typically fail to mention of the importance of sampling error on the precision of the local study estimate (Hunter & Schmidt, 2004; for recent exceptions, see Cronbach & Shavelson, 2004; Vacha-Haase, Henson, & Caruso, 2002). With small samples, the local estimate of reliability will usually be much less precise than a comparable estimate taken from the test manual or from a meta-analysis. There may be a tradeoff between precision and applicability of primary study estimates and meta-analytic reliability estimates. That is, the local estimate may be more applicable than is the meta-analytic estimate (because of the influence of research context), but the local estimate may be less precise than is the metaEstimating Reliability 4 analytic estimate (because of sampling error). Clearly we would like to know the precision of the local estimate and to be able to articulate what the tradeoff may be. It is also possible to blend the local and meta-analytic estimates. The current paper therefore describes and illustrates three things: 1. sampling variance and confidence intervals for Cronbach’s Alpha, 2. the relative precision of estimates from local studies and meta-analyses, 3. how to blend the local and meta-analytic information to create an optimal local estimate according to Bayesian principles (Lee, 1989; Brannick & Hall, 2003). Confidence Intervals for Alpha Cronbach’s alpha appears to be the most commonly reported estimate of reliability in the psychological research literature (Hogan, Benjamin, & Brezinski, 2000). Because it is an intraclass correlation, its sampling distribution is awkward and confidence intervals have only recently become available for it. However, it is quite important to report the precision of the estimate of alpha (that is, its standard error or confidence interval, Iacobucci & Duhachek, 2003) so that researchers can understand the likely magnitude of error associated with the estimate. An asymptotic (large sample) formula for the sampling variance of the function of Cronbach’s Alpha ) ˆ ( α α − n is shown by (Iacobucci & Dubachek, 2003, p. 480, Equation 2; van Zyl, Neudecker, & Nel, 2000, p. 276,Equations 20, 21): [ ] ) )( ( 2 ) )( ( ) ( ) 1 ( 2 2 2 2 3 2 2 j V j trV V tr trV Vj j Vj j k k Q ′ − + ′ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ′ − = (1) where k is the number of items that are added for form the composite whose reliability is indexed by alpha, j is a k x 1 column vector of ones, V is the covariance matrix of the Estimating Reliability 5 items (that is, the sample estimate of the population covariance matrix), and tr is the trace function (the sum of the diagonal elements of a matrix). The asymptotic 95 percent confidence interval is given by (Iacobucci & Dubachek, 2003): ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ ± n Q 96 . 1 α̂ (2) where n = (N-1). A small-scale simulation by van Zyl, Neudecker and Nel (2000) shows that the asymptotic estimate appears to yield reasonable results provided that N is a least 100. However, Yuan, Guarnaccia, and Hayslip (2003) recommended that bootstrap estimates be used to compute confidence intervals rather than asymptotic estimates. They based their recommendations on a comparison of methods using data from the Hopkins Symptom Checklist. Bootstrap estimates are computed by taking repeated samples of data with replacement from the study data to compute an empirical sampling distribution. The empirical sampling distribution is then inspected to see where the extremes of the distribution fall, that is, the bootstrap estimates allow for the calculation of an empirical confidence interval. SAS programs that can be used to compute both asymptotic and bootstrap confidence intervals for alpha can be found at http://luna.cas.usf.edu/~mbrannic/software/softdir.htm. The programs contain sample data from students who completed a questionnaire composed of some IPIP extroversion items. Based on responses of 100 people to the ten items, the estimated alpha is .89, with asymptotic confidence interval (95 percent) of .85 to .92. The bootstrap confidence interval (95 percent) ranges from .85 to .91, so there appears to be good agreement between the two different confidence intervals estimated by the asymptotic and bootstrap Estimating Reliability 6 methods in this case. Either method can be used to estimate the sampling variance of alpha for a local study. Unfortunately, both the asymptotic sampling variance and bootstrap methods require information that is not typically presented in journal articles. The asymptotic method requires the covariance matrix for the items, and the bootstrap method requires the raw data. Both methods are of interest to primary researchers, but meta-analysts typically do not have access to the required data. It is possible, however, to assume compound symmetry for the covariance matrix (that is, to assume that the covariance or correlation among all items is the same). Under the assumption of compound symmetry, it is possible to solve for the common covariance and then to use the asymptotic formula to estimate the sampling variance of alpha. Such a procedure is analogous to using the reported estimates of the mean and standard deviation to estimate the reliability (KR-21) of tests composed of dichotomous items. Under the assumption of compound symmetry, the expression for alpha becomes (van Zyl, Neudecker, & Nel, 2000, p. 272, Equation 3) ) 1 ( 1 − + = k k ρ ρ α (3) where k is the number of items and ρ is the common element in the covariance matrix (i.e., the correlation of each item with all other items). A little algebra allows us to solve for the common element, thus: ) 1 ( − − = k k α α ρ . (4) Estimating Reliability 7 For example, if alpha is .8 and there are 3 items, then the implied correlation matrix is 1 0.5714 0.5714 0.5714 1 0.5714 0.5714 0.5714 1 If the primary researcher reports alpha, the number of items, and the sample size, then the meta-analyst can find an approximate sampling variance and therefore confidence intervals for the alpha estimate. Precision of Estimates Although measurement experts routinely call for local estimates of reliability (and there is no real reason NOT to report them), measurement experts typically fail to note the relative precision of local and meta-analytic estimates of reliability. That is, they fail to note that the local estimates tend to contain much more sampling error than do metaanalytic estimates (see also Sawilowsky, 2000, for further aspects of the controversy about reporting local reliability estimates and their meaning). The second contribution of the current paper is to quantify the precision of the two estimates to allow an explicit comparison of the precision of the two different estimates. There can be no stock answer to the question of the relative precision of the estimated reliability in a given sample versus the mean reliability of a meta-analysis. For the local study, the uncertainty of the reliability depends chiefly on the number of items, the magnitude of the covariances, and the number of people[check this]. As the number of items, the size of the covariance, and the size of the sample all increase, the local estimate will become precise. The meta-analytic result will become precise as the local samples become precise and also as the number of studies in the meta-analysis increases. In both cases, the sampling variance can be used to index the precision of the estimate. Estimating Reliability 8 Let’s look at a single example. The data in the following table were copied from Thompson and Cook (2002). The study was a meta-analysis of reliability estimates for a survey assessing user satisfaction with library services across 43 different universities. Table 1 shows the alpha estimates for the total score (based on k=25 items), as well as the alpha estimates for one of the subscales (5 items), Information Access. The sample size for each sample is also reported. From the information given in Table 1, I computed an estimate of the common element (rho) based on the assumption of compound symmetry. I then calculated the estimated sampling variance for each study. For the Information Access subscale, the study sampling variance estimates ranged from .000144 to .001425. The mean sampling variance was .000593. The estimated population variance of the alpha coefficients (raw data) was.00148651. I divided that by 43 (the sample size) to find the variance of the mean (the squared standard error of the mean), which was .00003547. The variance estimates tell us about the precision of the estimates. If we compare the variances by forming ratios, we can get a single number that represents the relative precision of the estimates (relative efficiency in statistical terms). In this study, the mean of the meta-analysis was from about 4 to 40 times more precise than the individual study. The meta-analytic mean was about 17 times more precise than the average local study. Another way to consider the same issue is to examine the width of confidence intervals (the width is the difference between the maximum and minimum value of the confidence interval). For the individual studies, the width of the confidence interval, which was computed by taking four times the standard error, ranged from .048 to .151; the mean width was .095. The width of the confidence interval for the mean of the study reliabilities was .024, so the confidence intervals from the these studies tend to be on Estimating Reliability 9 average about four times wider than the confidence interval for the mean across studies. All of the confidence intervals tend to be rather small. Compare them to the confidence interval width for a correlation of .30 with N = 100, which has a value of .367, which is nearly four times larger than the average confidence width for these studies. The variance estimate for the correlation was computed by 1 ) 1 ( 2 2 2 − − = N r ρ σ , with ρ = .30 and N=100. (5) There are two main points to this exercise. First, it should be clear that the mean of the studies has less sampling error than do the individual studies. Second, even moderate alpha estimates can be rather precise. Blending Local and Meta-Analytic Estimates By allowing researchers to combine local and meta-analytic data, Bayesian estimates allow researchers with small samples to ‘borrow strength’ from the metaanalytic estimates in a statistically optimal way. The paper shows how to combine both overall or global estimates from a meta-analysis with a local estimate, and also to combine the output of a meta-analytic regression analysis with a local reliability estimate. For example, if size of company (or another continuous variable) has been shown to moderate the reliability of the measure, it is possible to calculate the estimated reliability for the current company from the meta-analytic regression and to combine that metaanalytic estimate with the local study data to yield a local ‘best’ estimate. Such a result is important in practical applications in which reliability influences the interpretation of the results. The approach described in this paper is based on the work of Brannick (2001) and Brannick and Hall (2003). Estimating Reliability 10 If we want to borrow strength from a meta-analysis to bolster our local study, we need estimates of the uncertainty of both the local estimate and for the meta-analytic estimate. For the local study, we will consider only sampling error as a source of uncertainty. The index we will use is the sampling variance estimate for alpha. Two different sources of uncertainty may apply to the meta-analysis, however. The first of these concerns the value of the mean of the meta-analysis. Because the meta-analysis is based on a finite number of observations (and our universe of generalization is typically infinite), the actual population mean cannot be known; it can only be estimated. The confidence interval for the meta-analytic mean quantifies the degree of uncertainty of this type. The variance associated with sampling error can be calculated in several ways in a meta-analysis. The simplest way is to calculate a sampling variance of the mean effect size is to do it just as you would for any raw data: k k S E ES Vs ) 1 ( ) ( 2